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Abstract. The Internet has recently encouraged the society to convert almost all its needs to elec¬ 
tronic resources such as e-libraries, e-cultures, e-entertainment as well as e-learning, which has 
become a radical idea to increase the effectiveness of learning services in most schools, colleges 
and universities. E-learning can not be completely featured and met without e-testing. However, 
in many cases e-testing tools are suitable just for traditional/theoretical knowledge testing, cov¬ 
ered by such items as questions, quizzes, matching boxes and other. The article “A Method for 
Automated Program Code Testing” tackles the lack of functions in e-testing systems and suggests 
e-assessment possibilities for students who study computer science, especially programming. The 
article analyzes the method that allows freely entering answers to questions, checking program 
syntax during the testing and enables automatic written code checking and evaluation. 

Keywords: e-learning, distance learning, e-testing tools, e-testing method, e-testing environment, 
e-tests, computer adaptive testing (CAT), computerized testing, computer-based testing, automatic 
code verification, program code testing. 


1. Introduction 

The Internet and information communication technologies are more and more extensively 
involved in daily learning processes all over the world. This growth has been stimulated 
in part by the interest of educators and trainers in the use of new, internet based and mul¬ 
timedia technologies, as well as by the recognition that traditional ways of organizing 
the education need to be reinforced by innovative methods intending to realize the funda¬ 
mental right of all people to learning. Electronic learning syllabus, discussion tools, video 
conferences and e-assessment tools seem to be trivial today and are placed in the centre of 
today’s learning environment. It is visible that distance and e-learning with various infor¬ 
mation systems and virtual learning environments have taken the first place within such 
processes. Therefore, the increasing importance of e-learning and the integration of in¬ 
formation systems in all areas of learning force teachers to understand and implement IS 
(information systems) and IS specialists to start thinking about more efficient and precise 
mechanisms for such system design and implementation, the development of new tools 
not just for making learning material more accessible but also for creating e-assessment 
tools. The use of computerized adaptive testing (CAT; Wainer, 2000) has substantially 
increased since it was first formulated in the 1970s (Meijer et al ., 1999). 
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Computerized testing greatly increases the flexibility of test management (Gibbons 
et al ., 2008). When an examinee administers a test via the computer, the computer can 
update the evaluation of the examinee’s ability after each item and then the ability to 
estimate can be used in the selection of subsequent items. With the right item bank and a 
high variance of examinee’s ability e-test can be more efficient than a traditional paper- 
and-pencil test (Gierl et al., 2008). 

In most cases e-testing tools are suitable just for traditional/theoretical knowledge 
testing while providing such items as questions, quizzes, matching boxes and other 
(Tonidandel et al., 2002). Nowadays suggested CATs are not applicable for all subjects 
and skills. Most CATs are based on an item-response theory model, yet the item response 
theory is not applicable to all skills and item types. While analyzing advantages and dis¬ 
advantages of CATs, Rudner (1998) claims that CATs require careful item calibration. 
The item parameters used in a paper-and-pencil testing may not always be applied and 
suitable in a computer adaptive test. Computer science tests especially related with pro¬ 
gramming fall into that category as well. A paradox, however, can be observed that in 
this information-communication technology age where computer programs are well and 
rapidly developed, students who learn programming are still manually tested. 

Thus, this article deals with e-testing method that allows to freely enter answers to 
questions, check program syntax during the testing and enables automatic written code 
checking and evaluation. The method will be used for students, studying computer sci¬ 
ence, to test their practical abilities of programming. 


2. E-Testing Today 

The migration of standardized tests from traditional paper-and-pencil administration 
to computerized adaptive administration offers many potential benefits, including in¬ 
creased measurement efficiency, immediate scoring, and more frequent administration 
dates (Harmes et al., 2001). 

Paper-and-pencil tests (traditional testing) are typically ’’fixed-item” tests in which the 
examinees answer the same questions within a given test booklet. Computerized testing 
offers a number of options for timing and formatting. Therefore, it has the potential to ac¬ 
commodate a wider range of item types. Tests are individually paced so that the examinee 
does not have to wait for others to finish before going on to the next section. Self-paced 
administration also offers extra time for the examinees that need it, potentially reducing 
one source of test anxiety (Georgiadou et al., 2006; Bergstrom, 1992). 

E-testing online assessments bring the best outcome through the reliable test engine 
which contains a test that reached the standards which went through the test creation 
process focusing on the reusability of the elements or big parts of tests (Brusilovsky 
et al., 2002). E-testing not only reduces time but also plays a big role while helping the 
teacher to check students’ knowledge and practical skills. 

During the semester students practically work, studying algorithms and writing pro¬ 
grams’ code. Testing of student knowledge only with standard tests is not enough because 
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Fig. 1. A typical knowledge control process: a) a testing process without any tool; b) the e-testing process. 


reading the question and understanding answers do not mean that student knows how to 
use his knowledge in a practical way (Tonidandel et al ., 2002). Therefore, most of the 
tests students are created by writing programs and checking them manually. A typical 
knowledge control process without and with e-testing environment from the lecturers’ 
point of view is illustrated in Fig. 1 and includes several phases. 

The difference seems to be minimal as the correction of students’ works is eliminated, 
but in fact this task takes about 75% of all the time (Hardre et al., 2006). But time saving is 
not the only advantage of e-testing system. E-testing tools present more useful functions: 
no manual evaluation is needed and statistics on question answering, student reports and 
many other things are provided. 

E-test is one of assessment modes, administered on the computer, intended to mea¬ 
sure the test-takers’ or students’ knowledge, skills and aptitudes. Knowledge is a set of 
theoretical information acquired on the subject and practical capability to apply that the¬ 
oretical knowledge in learner’s life (Brusilovsky et al., 1999). 

Existing learning management systems (LMS), suitable for e-learning, such as Black¬ 
board Learning System Vista, Moodle, ATutor and other offer a wide range of tests 
made of questionnaires, quizzes, crosswords, multiple-choices, free responses and etc. 
But most of them propose tests and quizzes made just from questions and few answers 
(Fig. 2a). 

Testing types, mentioned above, have become a popular way to assess the learner, 
check his/her knowledge; however, it has some disadvantages as well: 

• All possible answers both correct and incorrect are visible which allows students 
to guess the right answer even if he/she does not know that subject at all. 

• This type of knowledge assessment checks only the understanding of information 
(theoretical knowledge assessment) and capability to read and think. The practi¬ 
cal usage of theoretical information, which is a very important aspect, remains 
untouched. 
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class A { 

void say() {System.out.println( M Parent");} 

} 

class B extends A { 

void say() {System.out.println("Child M );} 
public static void main (String a[]) { 

A obj = new B(); 
obj.say(); 

} 

} 


What will be printed after the program 
code execution? 

a) Parent 

b) Child 

c) Nothing 

d) Code will not compile 


class A { 

void say() {System, out. println("Parent");} 

} 

class B extends A { 

void say() {System, out. println("Child");} 
public static void main (String a[]) { 

A obj = new B(); 
obj.say(); 

} 


What will be printed after the program code 
execution? 


a) 


b) 


Fig. 2. E-test types: a) question with answers, b) question with free entered answer. 


Some e-test types suggest possibility for students to enter free responses to questions 
(Fig. 2b). These answers are automatically compared by computer in straight equivalence 
with the right ones: 

• This kind of testing is a better way to check knowledge than allow choosing an¬ 
swers from given possibilities because answers are invisible and students need to 
think more. 

• Formulation of the answer depends on students’ ability to express his/her thoughts, 
the understanding of the subject and the topic; therefore, the comparison one by 
one does not function properly. 

Both ways leave the practical aspect away. But when speaking about computer sci¬ 
ence and information technologies, especially programming, it is obvious that testing 
of practical knowledge is essential. So there is a need for automated tests and program 
testing modules that help to reduce and ease the amount manual work, carried out by 
lecturers. 

A lot of work has already been done by helping people to study programming lan¬ 
guages and not only by reading books and finding information and examples over the 
Internet. W3Schools offer different materials on studying programming technologies. It 
offers Try-It-Yourself demos where it is possible to read the chapter and later try to change 
standard code examples for some programming languages like HTML 1 , SQL 2 , CSS 3 . It 
is very useful for students and it was great, if such experience was transferred into the 
world of tests. 


^TML Examples. URL: http: //www. w3schools . com/html/html_examples . asp. 

2 SQL Try It. URL: http : / /www. w3schools . com/sql/sql_tryit. asp. 

3 CSS Examples. URL: http: / /www. w3schools . com/Css/css_examples . asp. 
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3. Designing E-Testing Method 

Carrying out the research and teaching students who study computer science it was ob¬ 
served that tests where students can enter the program code and the code is automatically 
checked could be very attractive and useful, because fast answering is possible as well 
as the reduction of teacher’s work. This also allows enabling automatic checking of the 
code syntax’s errors and provides opportunities to find and correct code errors because 
they are not essential and concentrate on algorithms. 

The strongest part of such tests is a possibility to write program code in personal 
student style and check just the correctness of the result but not the way it is coded. This 
provides test’s flexibility. However, this is only possible with programming languages 
where coded functions can output some results, like SQL query results, or C, JAVA, PHP 
function’s returning values. Then the way the program code is written is not important as 
the emphasis is placed on the fact if the function provides correct answers with the test 
data. 

Such tests face the problems indicated below: 

• Result checking may determine wrong program code writing, using forbidden tech¬ 
niques or hard-coding the results. 

• It is very hard to check the correctness of the program when the program code does 
not generate the exact output. For instance, checking written HTML code needs 
sophisticated code analysis which will not be discussed in this article. 

This article tackles only the first problem indicated, which can be solved by entering 
allowable and disallowable keywords for the entered program code and checking answers 
not for syntax correctness but also for using such keywords. 

The second problem will not be analyzed in this article because the main purpose of 
the paper seeks to concentrate on programming languages that maintain functions which 
return exact results. 


4. Algorithm Implementation 

Automatic code verification needs different e-test question structure which allows not just 
entering concrete answers but also encapsulating answers into questions, executing and 
obtaining some answer that can be checked. 

The next figure (Fig. 3) presents alternative question structure, which is different from 
a standard quiz question. 

Proposed question structure allows entering full program code and hiding some parts 
that must be rewritten by a student or serves like extra cases for avoiding hard-coding. 
Keywords that are allowed or forbidden to be used in student answers need to be consid¬ 
ered as well. 

This section presents an example that illustrates the approach and mentions a possi¬ 
bility to solve the problem by showing how code verification can be carried out for the 
code that can return the answer. A test question in Fig. 4 is a fragment with JAVA program 


source. 
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Question 







Answer 1 






Answer n 







Program code 
Code that needs to be 
written by student 

Extra cases that need to 
be hidden 


Extra keywords that must be 
used in code 


Extra keywords that can not 
be used in code 


a) b) 

Fig. 3. Question structure meta-model: a) standard question; b) proposed question. 


class A { 

/* Method orders String characters in alphabet order */ 
String orderStr (String s) { 


/* To Do: Program code goes here */ 


} 

} 

class B { 

public static void main (String a[]) { 

A obj = new A(); 

if (!“acd”.equals(obj.orderStr(new String (“cda”)))) { 
System.out. printin(“False”); 
return; 

} 


/* Hidden cases... */ 
System.out.println(“T rue”); 

> 

> 


Fig. 4. Question example. 


In this case a student gets a fragment of the program with a code which is already writ¬ 
ten. He/she has to read the code, understand it and extend the program with a predefined 
function. He/she also needs to write a class method for entering letters in a text area in 
the right order. Later a student sees some test cases that are intended for written function 
testing. The function has to correspond to the test cases. There are some hidden cases that 
a student does not see as the test cases serve as a protection against hard-coding. In this 
particular case a loop sentence FOR and a condition sentence IF can be used, and SORT 
function is forbidden. 
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Fig. 5. Program code execution on the server. 
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Fig. 6. Question checking a) standard question; b) proposed question. 


The code execution on the server will involve saving student answer, which is his 
program, compiling the code with JAVA compiler (javac) and using Java Virtual Machine 
(JVM) from one side and Java-Bridge extension from PHP (see Fig. 5). 

Checking and submitting answers is completed using two steps (Fig. 6). Carrying out 
the first step, the code is checked for exceptions (forbidden/allowable keywords, using 
system and file functions) and syntax errors. Checking syntax knowledge refers to the 
ability to fix errors within the written program code. There is no need to test student’s 
syntax knowledge because all IDE provides code syntax checking. After that, the student 
can fix his/her errors and submit the functioning code for the evaluation. 
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Table 1 

Decision table for answer evaluation 


Conditions 


Program compiles (3 evaluation points) 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

N 

N 

N 

N 

N 

N 

N 

N 

Forbidden keywords are used 

Y 

Y 

Y 

Y 

N 

N 

N 

N 

Y 

Y 

Y 

Y 

N 

N 

N 

N 

Only right keywords are used (3 evaluation points) 

Y 

Y 

N 

N 

Y 

Y 

N 

N 

Y 

Y 

N 

N 

Y 

Y 

N 

N 

Result is correct (4 evaluation points) 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Actions 

















Allow program bug fixing 
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Tell to remove forbidden keywords 
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Evaluation is possible 
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Point 9 
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Point 10 
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Result 

















Sum 

0 

0 

0 

0 

10 

6 

7 

3 

3 

3 

0 

0 

0 

0 

0 

0 


To eliminate question’s evaluation problem when just two possible results are avail¬ 
able, the decision table (Table 1) is proposed to make set of results larger. In this case 
four evaluation conditions are offered and all possible cases are listed. Decision table 
allows planning actions that may be done according those conditions and also evaluate 
every condition that is satisfied by the student. Three conditions are evaluated with points, 
only if forbidden keywords are used evaluation must be stopped because of the possible 
hacking. Obtained result range is wider with decision table. 


5. Security 

Server scripting languages like PHP, ASP or JSP are possible to be used for the imple¬ 
mentation of tests. Such languages possess build in functions that allow executing C, 
JAVA or their own program codes and get an answer. There are many C or JAVA compil¬ 
ers available in the Internet and those languages are substantial seeking to check student 
knowledge on algorithm creation. 
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In all cases security remains the open question because a written program code may 
be harmful to e-testing server, that is why secure program execution becomes of key im¬ 
portance. Most work can be carried out during the phase of exception checking, because 
the checking of system usage and file functions must be limited. The next phase involves 
code saving and execution in a web safe directory, which is not reachable for web users. 


6. Conclusions 

E-testing has numerous advantages over traditional paper-and-pencil testing; however, it 
brings some challenges as well. The development of e-testing tools has become an im¬ 
portant issue in the development of e-learning systems. This article analyzes the method 
for the creation of e-testing tools. The suggested method allows creating and completing 
tests for computer science students. It is also suitable for checking and evaluating test 
questions with freely entered program code, automatic checking of syntax correctness 
and the usage of right keywords. 

This type of testing is a helpful tool for teachers as it reduces their work and provides 
students with evaluation of their answers. Moreover, e-testing would be a useful tool for 
employers, selecting and hiring programmers because this method allows testing their 
computer science skills, abilities and practical knowledge that is of key importance in 
this specialty. 
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Automatizuoto programos kodo testavimo metodas 

Sigitas DR4SUTIS, Vida MOTEKAITYTE, Algirdas NOREIKA 

Informacines komunikacines technologijos, interneto galimybes ir prieigos taskp didejimas 
skatina visuomene perkelti dali gyvenimiskp veiklp \ virtuali3 erdve, todel gana sparciai kuriasi 
e. bibliotekos, e. kultura, e. pramogos ir, zinoma e. mokymas, kuomet siekiama pagerinti moky- 
mosi efektyvum^ mokyklose, kolegijose, universitetuose. Siuolaikinis e. mokymas negali buti su- 
vokiamas be e. testavimo, taciau daugeliu atvejp testavimo irankiai yra dazniausiai pritaikomi ir 
naudojami tik standartinems/teorinems zinioms patikrinti, panaudojant klausimus, apklausas, su- 
lyginimus, pasirinkimus vien^/daug is kelip variantp ir panasiai. Siame straipsnyje aptariama e. 
testavimui skirtp sistemp funkcinip galimybip stoka ir pasiulomas metodas besimokanciiyu, studi- 
juojanciq informatik^, zinioms ivertinti. Pasiulytu zinip patikrinimo metodu sudaromos galimybes 
Jvesti laisvos formos atsakym^ (programos fragment^), patikrinti programinio kodo sintakse bei 
ivertinti parasytos programos teisingum^. 



